FedShakespeare

FedShakespeare#

class fl_sim.data_processing.FedShakespeare(datadir: str | Path | None = None, seed: int = 0, **extra_config: Any)#

基类：FedNLPDataset

Federated Shakespeare dataset.

Shakespeare dataset is built from the collective works of William Shakespeare. This dataset is used to perform tasks of next character prediction. FedML [1] loaded data from TensorFlow Federated (TFF) shakespeare load_data API [2] and saved the unzipped data into hdf5 files.

Data partition is the same as TFF, with the following statistics.

DATASET

TRAIN CLIENTS

TRAIN EXAMPLES

TEST CLIENTS

TEST EXAMPLES

SHAKESPEARE

715

16,068

715

2356

Each client corresponds to a speaking role with at least two lines.

参数:

  • datadir (Union[str, pathlib.Path], optional) -- The directory to store the dataset. If None, use default directory.

  • seed (int, default 0) -- The random seed.

  • **extra_config (dict, optional) -- Extra configurations.

引用

property candidate_models: Dict[str, Module]#

A set of candidate models.

char_to_id(char: str) → int[源代码]#

Convert a character to an integer index.

property doi: List[str]#

DOI(s) related to the dataset.

evaluate(probs: Tensor, truths: Tensor) → Dict[str, float][源代码]#

Evaluation using predictions and ground truth.

参数:
返回:

Evaluation results.

返回类型:

Dict[str, float]

get_dataloader(train_bs: int | None = None, test_bs: int | None = None, client_idx: int | None = None) → Tuple[DataLoader, DataLoader][源代码]#

Get local dataloader at client client_idx or get the global dataloader.

参数:

  • train_bs (int, optional) -- Batch size for training dataloader. If None, use default batch size.

  • test_bs (int, optional) -- Batch size for testing dataloader. If None, use default batch size.

  • client_idx (int, optional) -- Index of the client to get dataloader. If None, get the dataloader containing all data. Usually used for centralized training.

返回:

get_word_dict() → Dict[str, int][源代码]#

Get the word dictionary.

id_to_word(idx: int) → str[源代码]#

Convert an integer index to a character.

preprocess(sentences: Sequence[str], max_seq_len: int | None = None) → List[List[int]][源代码]#

Preprocess a list of sentences.

参数:

  • sentences (Sequence[str]) -- List of sentences to be preprocessed.

  • max_seq_len (int, optional) -- Maximum sequence length. If None, use default sequence length.

返回:

List of tokenized sentences.

返回类型:

List[List[int]]

property url: str#

URL for downloading the dataset.

view_sample(client_idx: int, sample_idx: int | None = None) → None[源代码]#

View a sample from the dataset.

参数:

  • client_idx (int) -- Index of the client on which the sample is located.

  • sample_idx (int) -- Index of the sample in the client.

返回类型:

None

property words: List[str]#

Get the word list.

目录